Environmental noise can cause an exponential reduction in the mean time to extinction (MTE) of an isolated population. We study this effect on an example of a stochastic birth-death process with rates modulated by a colored Gaussian noise. A path integral formulation yields a transparent way of evaluating the MTE and finding the optimal realization of the environmental noise that determines the most probable path to extinction. The population-size dependence of the MTE changes from exponential in the absence of the environmental noise to a power law for a short-correlated noise and to no dependence for long-correlated noise. We also establish the validity domains of the limits of white noise and adiabatic noise.